Skip to content

[feature](catalog) Add MongoDB session read policies - #70493

Open
Gabriel39 wants to merge 1 commit into
apache:masterfrom
Gabriel39:dev/mongodb-session-read-policy
Open

Gabriel39 wants to merge 1 commit into
apache:masterfrom
Gabriel39:dev/mongodb-session-read-policy

Conversation

@Gabriel39

Copy link
Copy Markdown
Contributor

Background

The native MongoDB catalog sends read preference and read concern to the BE in each scan descriptor. Today those values come only from catalog properties or the connection URI, while metadata discovery uses its own primary/default policy.

What problem does this PR solve?

Issue Number: #70251
Related PR: #70327 (native reader and catalog read policies)
PR Type: Feature

Problem Summary:

Users cannot choose MongoDB routing and visibility for an individual session or query without changing the shared catalog. Add mongodb_read_preference and mongodb_read_concern, supporting SET and SET_VAR, and pin their effective values in each scan descriptor. Both default to inherit, preserving existing catalog/URI behavior. Most of the diff adds tests and generated regression expectations.

Details

  • Change map: 70 production lines changed, 281 test-source lines changed, 62 generated expectation lines, and 38 documentation lines changed.
  • Scope: Data-read policies for native MongoDB scans. Preference supports inherit, primary, primaryPreferred, secondary, secondaryPreferred, and nearest; concern supports inherit, local, majority, and available. Values are case-insensitive. Snapshot and linearizable reads remain unsupported.
  • Design: Register validated, forwarded session variables and resolve them through the existing connector session-property snapshot. Reuse the existing scan descriptor and BE collection policy application. Session overrides do not mutate catalog configuration, metadata clients, or transport pool identity. URI tags, maximum staleness, and hedge options remain effective; an incompatible primary override fails during planning without exposing the URI.
  • Usage: SET mongodb_read_preference='secondaryPreferred'; SET mongodb_read_concern='local'; or SELECT /*+ SET_VAR(mongodb_read_preference='primary', mongodb_read_concern='majority') */ .... Set either variable to inherit to use its catalog/URI policy again.
  • Default behavior: inherit independently resolves each policy from URI options first, then catalog defaults. Queries without overrides retain their existing behavior.
  • Compatibility: No connector SPI, Thrift, BE code, or storage-format changes. Metadata discovery and collection-identity verification retain their existing policies. Read concern does not establish a snapshot shared by scans.

Testing

All checks below ran locally against MongoDB 8.0.20, with the new FE and the existing ASAN native BE:

  • Red/green: the new scan-plan tests first failed because overrides were ignored, and the new session tests first failed because the variables were unregistered; both passed after implementation.
  • run-fe-ut.sh --run with SessionVariableMongoReadPolicyTest, VariableMgrExportTest, MongoScanPlanTest, and MongoCatalogPropertiesTest: passed. The three new session tests cover defaults, supported/unsupported values, connector snapshots, isolation, and forwarding.
  • mvn -f fe/pom.xml -pl :fe-connector-mongodb -am test: 35 connector tests passed, including URI precedence, independent policy inheritance, incompatible routing criteria, and stable transport configuration identity.
  • build.sh --fe -j12: passed. FE core and MongoDB connector Checkstyle: zero violations.
  • Generated test_mongodb_read_policy.out using the regression runner, checked its results against the synthetic fixture, then independently reran test_mongodb_native and test_mongodb_read_policy: 2 suites passed, 0 failed/fatal/skipped. Covers SET/SET_VAR, all supported modes/levels, invalid values, failed-hint restoration, URI policy inheritance, and recovery.
  • Additional replica-set integration: source profiler verified primary/secondary selection and the actual read concern for 8 sequential reads and 32 concurrent reads across 4 isolated sessions. Also verified URI tags with a mode override and policy restoration after a failed hint. This was an additional local acceptance check; the committed SQL suites use the existing standalone fixture.

Release note

Add mongodb_read_preference and mongodb_read_concern session variables for native MongoDB queries, with statement-level SET_VAR support. Both default to inherit to retain the catalog's existing read policy.

Check List (For Author)

  • Behavior changed: Yes (see Details; explicit overrides now affect native MongoDB data reads).
  • Does this need documentation: Yes; the connector README is updated in this PR.
  • Pre-submission status: Ready. Local checks passed; CI is pending.

AI Disclosure

  • Tool: Codex (GPT-6).
  • Generated: Implementation, tests, connector documentation, and this description.
  • Human review: Maintainer review pending; the agent completed a full diff self-review and local validation.

### Background

The native MongoDB catalog sends read preference and read concern to the BE in each scan descriptor. Today those values come only from catalog properties or the connection URI, while metadata discovery uses its own primary/default policy.

### What problem does this PR solve?

Issue Number: apache#70251
Related PR: apache#70327 (native reader and catalog read policies)
PR Type: Feature

Problem Summary:

Users cannot choose MongoDB routing and visibility for an individual session or query without changing the shared catalog. Add `mongodb_read_preference` and `mongodb_read_concern`, supporting `SET` and `SET_VAR`, and pin their effective values in each scan descriptor. Both default to `inherit`, preserving existing catalog/URI behavior. Most of the diff adds tests and generated regression expectations.

### Details

- **Change map:** 70 production lines changed, 281 test-source lines changed, 62 generated expectation lines, and 38 documentation lines changed.
- **Scope:** Data-read policies for native MongoDB scans. Preference supports `inherit`, `primary`, `primaryPreferred`, `secondary`, `secondaryPreferred`, and `nearest`; concern supports `inherit`, `local`, `majority`, and `available`. Values are case-insensitive. Snapshot and linearizable reads remain unsupported.
- **Design:** Register validated, forwarded session variables and resolve them through the existing connector session-property snapshot. Reuse the existing scan descriptor and BE collection policy application. Session overrides do not mutate catalog configuration, metadata clients, or transport pool identity. URI tags, maximum staleness, and hedge options remain effective; an incompatible primary override fails during planning without exposing the URI.
- **Usage:** `SET mongodb_read_preference='secondaryPreferred'; SET mongodb_read_concern='local';` or `SELECT /*+ SET_VAR(mongodb_read_preference='primary', mongodb_read_concern='majority') */ ...`. Set either variable to `inherit` to use its catalog/URI policy again.
- **Default behavior:** `inherit` independently resolves each policy from URI options first, then catalog defaults. Queries without overrides retain their existing behavior.
- **Compatibility:** No connector SPI, Thrift, BE code, or storage-format changes. Metadata discovery and collection-identity verification retain their existing policies. Read concern does not establish a snapshot shared by scans.

### Testing

All checks below ran locally against MongoDB 8.0.20, with the new FE and the existing ASAN native BE:

- Red/green: the new scan-plan tests first failed because overrides were ignored, and the new session tests first failed because the variables were unregistered; both passed after implementation.
- `run-fe-ut.sh --run` with `SessionVariableMongoReadPolicyTest`, `VariableMgrExportTest`, `MongoScanPlanTest`, and `MongoCatalogPropertiesTest`: passed. The three new session tests cover defaults, supported/unsupported values, connector snapshots, isolation, and forwarding.
- `mvn -f fe/pom.xml -pl :fe-connector-mongodb -am test`: 35 connector tests passed, including URI precedence, independent policy inheritance, incompatible routing criteria, and stable transport configuration identity.
- `build.sh --fe -j12`: passed. FE core and MongoDB connector Checkstyle: zero violations.
- Generated `test_mongodb_read_policy.out` using the regression runner, checked its results against the synthetic fixture, then independently reran `test_mongodb_native` and `test_mongodb_read_policy`: 2 suites passed, 0 failed/fatal/skipped. Covers SET/SET_VAR, all supported modes/levels, invalid values, failed-hint restoration, URI policy inheritance, and recovery.
- Additional replica-set integration: source profiler verified primary/secondary selection and the actual read concern for 8 sequential reads and 32 concurrent reads across 4 isolated sessions. Also verified URI tags with a mode override and policy restoration after a failed hint. This was an additional local acceptance check; the committed SQL suites use the existing standalone fixture.

### Release note

Add `mongodb_read_preference` and `mongodb_read_concern` session variables for native MongoDB queries, with statement-level SET_VAR support. Both default to `inherit` to retain the catalog's existing read policy.

### Check List (For Author)

- Behavior changed: Yes (see Details; explicit overrides now affect native MongoDB data reads).
- Does this need documentation: Yes; the connector README is updated in this PR.
- Pre-submission status: Ready. Local checks passed; CI is pending.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: Implementation, tests, connector documentation, and this description.
- Human review: Maintainer review pending; the agent completed a full diff self-review and local validation.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all eight changed paths at de95c7b. No new actionable findings and no existing blocking inline comments. Review converged after two rounds, including independent session, connector, and risk-focused reviews and a final challenge of this frozen payload.

Critical checkpoint conclusions:

  • Goal and evidence: SET and SET_VAR policies independently inherit or override catalog/URI defaults and are pinned in Mongo scan descriptors. New FE and connector unit tests and the regression suite cover the intended behavior; test sources were inspected, not executed.
  • Scope and reuse: the change is focused and uses the existing session-property snapshot, scan provider, descriptor, and native reader.
  • Concurrency and locks: each scan retains its own session map, descriptor and collection policy; pooled clients are leased exclusively. No new shared mutable policy, lock order or work under metadata locks was introduced.
  • Lifecycle: range descriptors are deep-copied, cursor/collection cleanup precedes pool return, and the existing failure path closes acquired resources. SET_VAR restoration runs in the executor's finally path, including planning errors. No new static initialization or resource lifetime was introduced.
  • Configuration: both variables default to inherit, validate supported values case-insensitively, update without restart, and participate in generic session export and FE forwarding.
  • Compatibility: existing Mongo descriptor fields are reused; no connector SPI, function symbol, storage format or new wire field changes. Missing session keys preserve inheritance. Existing native policy consumers accept the canonical values.
  • Thrift/protobuf IDs: no gensrc files changed; branch ID comparison is inapplicable.
  • Parallel paths: collection, view and time-series scans use the same provider. Prepared external queries replan, forwarded queries carry the variables, and Mongo tables are excluded from SQL result caching. Metadata discovery and collection identity verification retain independent policies.
  • Conditional checks and errors: unsupported policies fail rather than fall back. The primary/URI routing-criteria check matches the native consumer's constraints; inherited criteria remain attached to the scan (checked against the pinned C driver and Java driver implementations).
  • Test coverage: inspected defaults, supported and rejected values, case handling, URI precedence, primary conflicts, immutable snapshots, transport identity, hints and restoration. The committed standalone fixture cannot establish replica selection or concurrent replica behavior.
  • Expected results: the fixture's four rows have seq values 1 through 4, matching count=4 and sum=10. Variable and hint/restoration expectations match the SQL sequence; SHOW output is ordered and aggregate results have one row. Result generation was not independently reproduced.
  • Observability: variables are visible through SHOW/session inspection; existing changed-variable audit capture and native errors remain available. No new metric or high-volume log requirement was identified.
  • Transactions and persistence: no source transaction, EditLog protocol or failover behavior changes. Existing reflected session-variable serialization covers the added strings; read concern does not create a shared snapshot across scans.
  • Data writes and atomicity: this change only affects external reads; no write, delete-bitmap, rowset or Doris visible-version logic changes.
  • FE-to-BE propagation: canonical effective preference/concern values reach the existing TMongoScanDesc on each native range; constant folding and OLAP point-query paths do not read Mongo collections.
  • Performance and memory: added resolution does bounded local work per scan, without new I/O or row-loop work. No new long-lived allocation or BE memory ownership was introduced.
  • Other concerns and user focus: no additional focus was supplied. Both initial risk items were dismissed with concrete upstream/downstream evidence, and no code-level candidate remains unresolved.

Validation limits: static inspection only. No builds, tests, benchmarks, cluster operations or runtime reproduction were performed. Author-reported replica-set profiling and concurrency checks were not independently verified. These execution limits do not leave an uncovered portion of the required static review.

Review converged after 2 round(s).

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant